Image breathing correction systems and related methods

ABSTRACT

Methods of image breathing correction. Implementations may include capturing and displaying a first image to a user using a lens, an image sensor, memory, and a display included in a camera unit, adjusting a position of the lens, capturing a second image and storing data of the second image collected by the image sensor in the memory. The method may include determining a second position of the lens using a lens position sensor included in the camera unit and calculating a change in magnification from the first image to the second image using a processor and a breathing correction model. The method may include rescaling the data of the second image to generate corrected second image data using the processor and displaying the corrected second image to a user. The corrected second image may be substantially free from breathing effects when compared with the first image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of the earlier U.S.Utility Patent Application to Stern entitled “Image Breathing CorrectionSystems and Related Methods,” application Ser. No. 15/165161, filed May26, 2016, now pending, which claims the benefit of the filing date ofU.S. Provisional Patent Application 62/253,479, entitled “ImageBreathing Correction Systems and Related Methods” to Stern which wasfiled on Nov. 10, 2015, the disclosures of each of which are herebyincorporated entirely herein by reference.

BACKGROUND 1. Technical Field

Aspects of this document relate generally to camera imaging systems,such as systems involving digital image sensors.

2. Background

Many camera systems utilize a focus system. During a typical manual orautofocus sequence, images are captured from various lens positions inorder to determine the best focus position. The change in lens positionresults in a change in magnification of the image. This change inmagnification of the image as the camera focus changes is visible to theuser and is often referred to as “breathing.”

SUMMARY

Implementations of a method of generating a breathing correction modelmay include: providing a Field of View/Distortion (FOV/D) test image,placing the FOV/D test image in the field of view of a lens included ina camera unit including an image sensor and a lens position sensor andmoving the lens to a first focus distance. The method may also includestoring a first image of the FOV/D test image in a memory included inthe camera unit using the image sensor and determining a first lensposition at the first focus distance using the lens position sensor. Themethod also may include generating a first calibrated Inverse FOV/D(IFOV/D) correction map by comparing FOV/D data of the first image withreference data stored in the memory associated with the FOV/D test imageusing a processor included in the camera unit. The method also mayinclude storing the first calibrated IFOV/D correction map in thememory, moving the lens to a second focus distance, and storing a secondimage of the FOV/D test image in a memory included in the camera unit.The method also may include determining a second lens position at thesecond focus distance using the lens position sensor and generating asecond calibrated IFOV/D correction map by comparing FOV/D data of thesecond image with reference data stored in the memory associated withthe FOV/D test image using the processor. The method may also includestoring the second calibrated IFOV/D correction map in the memory wherethe first calibrated IFOV/D correction map and second calibrated IFOV/Dcorrection map may form a breathing correction model.

Implementations of a method of generating a breathing correction modelmay include one, all, or any of the following:

Moving the lens to a third focus distance, storing a third image of theFOV/D test image in the memory included in the camera unit, anddetermining a third lens position at the third focus distance using thelens position sensor. The method may also include generating a thirdcalibrated IFOV/D correction map by comparing FOV/D data of the thirdimage with reference data stored in the memory associated with the FOV/Dtest image using the processor. The method may also include storing thethird calibrated IFOV/D correction map in the memory where the firstcalibrated IFOV/D, second calibrated IFOV/D correction map, and thirdcalibrated IFOV/D correction form the breathing correction model.

The lens position sensor may be a Hall Effect sensor.

The breathing correction model may be stored in the memory in a look uptable.

The FOV/D data of the first image and the FOV/D data of the second imagemay be calculated using a distortion model that is 3^(rd) order, 5^(th)order, or arctangent/tangent.

Implementations of a method of correcting for breathing effects forimages may include capturing and displaying a first image to a userusing a lens, an image sensor, memory, and a display included in acamera unit and adjusting a position of the lens. The method may alsoinclude capturing a second image with the lens and image sensor andstoring data of the second image collected by the image sensor in thememory. The method also may include determining a second position of thelens at the time of the capturing of the second image using a lensposition sensor included in the camera unit and calculating an IFOV/Dcorrection map for the second position of the lens using a processorincluded in the camera unit and a breathing correction model stored inthe memory. The method may also include applying the IFOV/D correctionmap to the data of the second image to generate corrected second imagedata using the processor, storing the corrected second image data in thememory, and displaying the corrected second image to a user using thedisplay. The corrected second image may be substantially free frombreathing effects when compared with the first image.

Implementations of a method of correcting for breathing effects forimages may include one, all, or any of the following:

The breathing correction model may include at least a first calibratedIFOV/D correction map and a second calibrated IFOV/D correction map.

Calculating the IFOV/D correction map for the position for the secondlens may further include interpolating the IFOV/D correction map usingthe second position of the lens and the at least first calibrated IFOV/Dcorrection map and the second calibrated IFOV/D correction map.

FOV/D data used to generate the at least first calibrated IFOV/Dcorrection map and the second calibrated IFOV/D correction map may becalculated using a distortion model that is 3^(rd) order, 5^(th) order,or arctangent/tangent.

The breathing correction model may be stored in a look up table.

The process or may be a graphics processor unit (GPU).

The lens position sensor may be a Hall Effect sensor.

Implementations of a method of correcting for breathing effects forimages may include capturing and displaying a first image to a userusing a lens, an image sensor, memory, and a display included in acamera unit, adjusting a position of the lens, capturing a second imagewith the lens and image sensor and storing data of the second imagecollected by the image sensor in the memory. The method may furtherinclude determining a second position of the lens at the time of thecapturing of the second image using a lens position sensor included inthe camera unit and calculating a change in magnification from the firstimage to the second image for the second position of the lens using aprocessor included in the camera unit and a breathing correction modelstored in the memory. The method may also include rescaling the data ofthe second image to generate corrected second image data using theprocessor, storing the corrected second image data in the memory, anddisplaying the corrected second image to a user using the display. Thecorrected second image may be substantially free from breathing effectswhen compared with the first image.

Implementations of a method of correcting for breathing effects forimages may include one, all, or any of the following:

The breathing correction model may include at least a first calibratedIFOV/D correction map and a second calibrated IFOV/D correction map.

Calculating a change in magnification from the first image to the secondimage may further include interpolating an IFOV/D correction map usingthe second position of the lens and the at least first calibrated IFOV/Dcorrection map and the second calibrated IFOV/D correction map.

FOV/D data used to generate the at least first calibrated IFOV/Dcorrection map and the second calibrated IFOV/D correction map may becalculated using a distortion model that is 3^(rd) order, 5^(th) order,or arctangent/tangent.

The breathing correction model may be stored in a look up table.

The processor may be a graphics processor unit.

The lens position sensor may be Hall Effect sensor.

The foregoing and other aspects, features, and advantages will beapparent to those artisans of ordinary skill in the art from theDESCRIPTION and DRAWINGS, and from the CLAIMS.

BRIEF DESCRIPTION OF THE DRAWINGS

Implementations will hereinafter be described in conjunction with theappended drawings, where like designations denote like elements, and:

FIG. 1 is flow chart illustrating a method of breathing correction;

FIG. 2 is a diagram showing a graph of an autofocus process using focusscores above captured images demonstrating the breathing effect causedby movement of the lens;

FIG. 3 is a block diagram of the components of a camera unit;

FIG. 4 illustrates an implementation of a Field of View/Distortion testimage.

DESCRIPTION

This disclosure, its aspects and implementations, are not limited to thespecific components, assembly procedures or method elements disclosedherein. Many additional components, assembly procedures and/or methodelements known in the art consistent with the intended image breathingcorrection system will become apparent for use with particularimplementations from this disclosure. Accordingly, for example, althoughparticular implementations are disclosed, such implementations andimplementing components may comprise any shape, size, style, type,model, version, measurement, concentration, material, quantity, methodelement, step, and/or the like as is known in the art for such imagebreathing correction systems, and implementing components and methods,consistent with the intended operation and methods.

Referring to FIG. 3, a block diagram of the components of a camera unit2 which form part or all of an image breathing correction system areillustrated. As illustrated, the camera unit 2 includes a lens 4 coupledto a body of the camera unit (not shown). The lens 4 is coupled to alens position sensor 6 which is coupled with a memory 8 included in thecamera unit 2. Operatively coupled with the memory 8 is a processor 10,an image sensor 12, and a display 14. In various implementations, theimage sensor is a digital image sensor. The processor may be a graphicsprocessor unit (GPU) or a central processing unit (CPU) in variousimplementations. The lens position sensor may be a Hall Effect sensor orother position sensing device.

A wide variety of image recording and storage devices may include thesecomponents. These include, by non-limiting example, camcorders,smartphone cameras, cell phone cameras, digital still cameras, mobilephones, digital cameras, video cameras, and other image capturingdevices. In various of these devices, the need to focus the image beingobserved through the lens 4 onto the image sensor 12 is accomplished bymanually or automatically moving the position of the lens 4 relative tothe image sensor 12 (or vice versa). In various camera units, the focusmay be accomplished by using an autofocus procedure. Depending on thestructure of the lens itself, the lens may be formed of multipleelements. In some devices like digital still cameras and camcorders, thefocus may be achieved by moving one or more elements of the lens itselfrather than the entire lens. In other devices, such as mobile phones,the entire lens (multi-element or not) may be moved to achieve focus.

The movement of the lens causes the magnification of the image beingviewed through the lens to change. Similar to the action of a magnifyingglass or microscope, movement of the lens (or a component thereof) tochange the focus also changes the magnification of the image. Because ofthis result, during an autofocus procedure, the image that is eventuallydisplayed in focus is most likely to be either larger or smaller inmagnification from the image that was initially viewed by the user onthe display 14 of the camera unit 2. This change in magnification duringfocusing is referred to as “breathing.” Conventional camera units do notcorrect for the breathing effect during focusing. For mobile phonedevices, complex lenses operated using MEMS devices or electro-opticalelements like liquid crystal lenses could be used to allow for movementof only one or a few pieces of the lens to minimize the breathingeffect. However, even minimizing the amount of the lens that must movedoes not, in itself, eliminate or correct for the breathing effect.

FIG. 2 illustrates an autofocus process used by a camera unit to put theobjects in view of the lens in a best focus. As illustrated in the graphabove, a set of images are initially recorded in sequence and data foreach is stored in the memory 8 and analyzed by the processor 10. Theanalysis for each image is the calculation of a focus score for theobjects visible in the lens. The process of calculation of the focusscore can take place using a variety of conventional methods andalgorithms familiar to those of ordinary skill in the art. As can beobserved, the autofocus process begins by first capturing a first image(Frame 1) at a first lens position and then causing the lens 4 to travelto a second lens position whereupon a second image (Frame 2) iscaptured. This process is iteratively repeated with the lens 4 moving tothird, fourth, and ultimately fifth positions where a value of the focusscore is maximized. Note that the size of the objects displayed in frame5 is larger than the size of the objects in frame 1, demonstrating how,through breathing, the magnification of the objects has changed. Whilebreathing has been explained in the context of autofocus, the sameeffect occurs when the focus is manually adjusted as well.

Methods of calculating a change in magnification based on the positionof the lens and rescaling the data of the in-focus image are disclosedherein. A wide variety of methods can be used to calculate the change inmagnification based on the position of the lens. For example, inparticular implementations, various locations on the objects in viewcould be identified in a first image taken using the image sensor 12through the lens 4 at a given lens position recorded by the lensposition sensor. The X/Y translation of the various locations on theobjects may then be determined by comparing the first image with thesecond in-focus image and the number/arrangement of pixels in the secondimage adjusted through interpolation and/or cropping to bring the sizeof the objects in the second image substantially equivalent to theirsize in the first image. Other methods may also be used, including theuse of field of view (FOV) and distortion (lens distortion, D) data forthe camera unit.

In particular implementations, the use of FOV/D data for the lens 4 mayallow for the accurate rescaling of the image. In variousimplementations, the use of FOV/D data for the lens 4 may also allow forrescaling while not correcting any distortion of the corrected imagesimultaneously. Referring to FIG. 1, a flow diagram of an implementationof a method of breathing correction is illustrated. As illustrated, thelens 4 is brought to a starting lens position (step 16) and a firstimage is captured by the image sensor 12. The data associated with thefirst image is stored in the memory 8. In various implementations of themethod, the first image is then displayed on the display 14 using thememory 8 and, in some implementations, the processor 10 (step 18). Thelens position sensor 6 is then used to measure the lens position whilethe processor 10 is used to calculate a focus score for the first image(step 20). The camera unit 2 then moves the lens 4 to a second lensposition where a second image is captured (step 22). The lens positionsensor 6 is then used to measure the second lens position and theprocess is used to calculate a focus score for the second image (step24). The processor then uses a breathing correction model stored in thememory and the data of the second image to calculate the change inmagnification of the image from the starting position (step 26). Theprocessor then conducts an image rescale operation to the data of thesecond image to correct for the magnification change (breathing effect,step 28). The camera unit 2 then may display the corrected second imagesubstantially free from breathing effects when compared with the firstimage on the display 14 (step 30). Since the rescale operation istypically a mathematical/algorithm based on a model, precisely exactcorrection may be difficult to obtain, but substantially effectivecorrection may be achieved, as will be appreciated by those of ordinaryskill in the art. Since changes in distortion occur naturally as thelens is moved from the first lens position to the second lens position,there is still a change in distortion between the first image and thesecond image. However, depending on the implementation, the rescalingprocess may correct for this distortion change (and any existingdistortion of the second image), may only rescale the second image, ormay simultaneously correct for distortion and rescale.

The breathing correction model may, in various implementations, becreated using FOV/D data of the lens 4 and image sensor 12 of the cameraunit 2. In various implementations, implementations a method ofgenerating a breathing correction model may be used that involvecreating of FOV/D data. The methods may include providing an FOV/D testimage and placing the FOV/D test image in the field of view of the lens4 (viewing the test image through the lens 4). FIG. 4 is an example ofan implementation of a FOV/D test image that could be used in variousmethod implementations. The method also includes moving the lens 4 to afirst focus distance and storing a first image of the FOV/D test imagein the memory 8 using the image sensor 12. The first lens position isdetermined using the lens position sensor 6. In particularimplementations, the lens position is determined relative to the imagesensor 12. The method also includes generating a first Inverse FOV/D(IFOV/D) correction map (first calibrated IFOV/D correction map) bycomparing FOV/D data of the first image with reference data stored inthe memory associated with the FOV/D test image using the processor 10.The reference data is preloaded into the memory prior to the taking ofthe first image, or is supplied to the memory 8 and/or the processor 10during generating of the IFOV/D correction map.

In various implementations, the FOV/D data of the first image iscalculated using a distortion model that is 3^(rd) order, 5^(th) order,or arctangent/tangent. The 3^(rd) order model is r_(u)=r_(d)+k₁r_(d) ³where r_(u) is the undistorted radius; r_(d) is the distorted radius.Both values of r are normalized to the center-to-corner distance. k₁=0for no distortion; k₁<0 for pincushion distortion; k₁>0 for barreldistortion. The 5^(th) order model is r_(u)=r_(d)+h₁r_(d) ³+h₂r_(d) ⁵.The arctangent/tangent models are r_(u)=tan (10p₁r_(d))/(10p₁); k₁>0(barrel distortion) and r_(u)=tan⁻¹(10p₁r_(d))/(10p₁); k₁<0 theseequations, p and h represent the distortion coefficients. Variousoptimization algorithms can be used to calculate the distortioncoefficients. In particular implementations, the algorithm may find thestraightest top and bottom bars and most even distances between the leftedges and the right edges of the squares located at the center row ofthe FOV/D test image. Residual error (square root of the sum of squaresof the errors) may be used to indicate how well the algorithm worked.The field of view data used may be either the angular field of view indegrees or the linear field of view.

Following calculation of the FOV/D data for the first image, the inverseFOV/D correction map is generated by creating a set of correctioncoefficients/values that, when applied to the data of the first image asoriginally captured, make the first image substantially the same inappearance as the FOV/D Test image. The calibrated inverse FOV/Dcorrection map for the first image can then be stored in the memory 8 aspart of the breathing correction model. In particular implementations,the IFOV/D correction map may be stored in a look up table in the memory8 as part of the model.

The method also includes moving the lens 4 to a second focus distanceand capturing and storing a second image of the FOV/D test image in thememory 8 included in the camera unit 2. The method also includesdetermining a second lens position at the second focus distance usingthe lens position sensor 6 and generating a second calibrated IFOV/Dcorrection map by comparing FOV/D data of the second image withreference data stored in the memory 8 associated with the FOV/D testimage using the processor 10. The method also includes storing thesecond IFOV/D correction map in the memory 8. The first and secondIFOV/D correction maps then form part or all of the breathing correctionmodel stored in the memory 8.

In various method implementations, the steps of moving the lens 4 to anew focus distance, capturing and storing an image of the FOV/D testimage in the memory, generating a calibrated IFOV/D correction map, andstoring the map in the memory as part of the breathing correction modelmay be repeated one or as many times as desired to build a breathingcorrection model. More than one FOV/D test image or other test imagesother than FOV/D test images may be used in various methodimplementations.

Methods of correcting for breathing effects for images may includecapturing and displaying a first image to a user using the lens 4, imagesensor 12, memory 8, and the display 14 of the camera unit 2 thenadjusting the position of the lens 4. Various implementations may alsoinclude storing the first image in the memory 8 and/or determining theposition of the lens 4 at the time of the capture of the first image.The method also includes capturing a second image with the lens 4 andimage sensor 12 and storing data of the second image collected by theimage sensor 12 in the memory 8. The method also includes determining afirst position of the lens 4 at the time of the capture of the secondimage using the lens position sensor 6. Method implementations includecalculating a change in magnification from the first image to the secondimage for the second position of the lens 4 using the processor 10 andthe breathing correction model stored in the memory 8 and rescaling thedata of the second image to generate corrected second image data usingthe processor.

In various method implementations, the breathing correction modelincludes at least the first and second calibrated IFOV/D correction map(IFOV/D correction map). Also, calculating a change in magnificationfrom the first image to the second image further includes interpolatingan IFOV/D correction map for the second image using the position of thelens at the time of the capturing of the second image and the at leastfirst calibrated IFOV/D correction map and the second calibrated IFOV/Dcorrection map. The calibrated IFOV/D correction maps may be generatedusing any method disclosed in this document.

The method may further include storing the corrected second image datain the memory and displaying the corrected second image to a user usingthe display where the corrected second image is substantially free frombreathing effects when compared with the first image. To the userobserving the second image in comparison with the first image, theobjects in the second image appear substantially the same size withoutmagnification even after the change in lens position. Implementations ofmethods of correcting for breathing effects may be used duringautofocusing sequences, but they may also be used in other situations,such as during manual operation of the camera by the user.

In various method implementations, correction of the image may takeplace in one or both axes. In implementations where one axis is used forcorrection (such as the x or y axis), the correction process may involveusing a one line buffer in the memory and applying the correction toform a corrected line of image data line by line at the same rate as thedata of the second image is being read from the image sensor. In variousimplementations, a double buffer (two line) may be used to allow theprocessor to process the previously stored line (t−1) while the currentline of data is being stored as it is read from the image sensor. Inother implementations, the one line buffer may be sufficient providedthe correction can be applied during the row blanking time of thebuffer. In other implementations, more than two lines may be used, orthe correction process may not take place until some, most, or all ofthe data of the second image has been read from the image sensor.

In places where the description above refers to particularimplementations of image breathing correction systems and implementingcomponents, sub-components, methods and sub-methods, it should bereadily apparent that a number of modifications may be made withoutdeparting from the spirit thereof and that these implementations,implementing components, sub-components, methods and sub-methods may beapplied to other image breathing correction systems and related methods.

What is claimed is:
 1. A method of correcting for breathing effects forimages comprising: capturing a first image using a lens, an image sensorand a memory comprised in a camera unit; capturing a second image withthe lens and image sensor and storing data of the second image collectedby the image sensor in the memory; determining a second position of thelens at the time of the capturing of the second image using a lensposition sensor included in the camera unit; calculating an inverseField of View/Distortion (IFOV/D) correction map for the second positionof the lens using a processor comprised in the camera unit and abreathing correction model stored in the memory, wherein the breathingcorrection model stored in the memory is calculated with a Field ofView/Distortion (FOV/D) test image; applying the IFOV/D correction mapto the data of the second image to generate corrected second image datausing the processor.
 2. The method of claim 1, wherein the breathingcorrection model comprises at least a first calibrated IFOV/D correctionmap and a second calibrated IFOV/D correction map.
 3. The method ofclaim 2, wherein calculating the IFOV/D correction map for the secondposition of the lens further comprises interpolating the IFOV/Dcorrection map using the second position of the lens and the at leastfirst calibrated IFOV/D correction map and the second calibrated IFOV/Dcorrection map.
 4. The method of claim 2, wherein FOV/D data used togenerate the at least first calibrated IFOV/D correction map and thesecond calibrated IFOV/D correction map was calculated using adistortion model that is one of 3rd order, 5th order, orarctangent/tangent.
 5. The method of claim 1, wherein the breathingcorrection model is stored in a look up table.
 6. The method of claim 1,wherein the processor is a graphics processor unit (GPU).
 7. The methodof claim 1, wherein the lens position sensor is a Hall Effect sensor. 8.A method of correcting for breathing effects for images comprising:capturing a first image using a lens, an image sensor and a memorycomprised in a camera unit; capturing a second image with the lens andimage sensor and storing data of the second image collected by the imagesensor in the memory; determining a second position of the lens at thetime of the capturing of the second image using a lens position sensorincluded in the camera unit; calculating a change in magnification fromthe first image to the second image for the second position of the lensusing a processor comprised in the camera unit and a breathingcorrection model stored in the memory, wherein the breathing correctionmodel stored in the memory is calculated with a Field of View/Distortion(FOV/D) test image; and rescaling the data of the second image togenerate corrected second image data using the processor.
 9. The methodof claim 8, wherein the breathing correction model comprises at least afirst calibrated IFOV/D correction map and a second calibrated IFOV/Dcorrection map.
 10. The method of claim 9, wherein calculating a changein magnification from the first image to the second image furthercomprises interpolating an IFOV/D correction map using the secondposition of the lens and the at least first calibrated IFOV/D correctionmap and the second calibrated IFOV/D correction map.
 11. The method ofclaim 9, wherein FOV/D data used to generate the at least firstcalibrated IFOV/D correction map and the second calibrated IFOV/Dcorrection map was calculated using a distortion model that is one of3rd order, 5th order, or arctangent/tangent.
 12. The method of claim 8,wherein the breathing correction model is stored in a look up table. 13.The method of claim 8, wherein the processor is a graphics processorunit (GPU).
 14. The method of claim 8, wherein the lens position sensoris a Hall Effect sensor.